Automatic Item Generation via Frame Semantics:

These materials are an unpublished, proprietary work of ETS. Any limited distribution shall not constitute publication. This work may not be reproduced or distributed to third parties without ETS's prior written consent. Submit all requests through There is an increasing interest in the use of Automatic Item Generation (AIG) in educational assessment, concomitant with the development of technologies which have brought delivery of test content by computers into the mainstream. Automatic item generation-the practice of creating assessment items algorithmically-can be motivated in part by a number of obvious practical advantages (cf. Bennett in press). AIG can speed or even partially automate the development of new items, it can provide similar items at the same level of difficulty, thus improving test security, and it can support adaptive testing by providing similar items that vary systematically in difficulty. Given the attractiveness of these goals, much recent research has focused upon evaluating the extent to which these promises can be met (cf. However, automatic item generation obviously and critically depends upon well-founded models of the cognitive abilities underlying performance. Where such models are lacking, generation algorithms can have heuristic usefulness only. Much of the recent progress in AIG has come in areas where there are well-founded cognitive theories to support the development of reasoning (Newstead et al. 2002). It is thus important to approach AIG in the light of theories of test construction which ground item design upon an evidentiary basis, such as Evidence Centered Design (ECD), cf. Mislevy et al. 2002. It is not particularly surprising to note that applications of AIG have been concentrated in areas where the underlying cognitive domain is restricted in content and highly structured (e.g., various forms of abstract reasoning and mathematics.) Progress in automatic generation of verbal 1 item types has been much more limited, due to the openness of content and the considerable complexity of natural language. For passage-based verbal items, there is a strong preference for developing naturalistic items based upon actually published materials, and the most productive approaches have focused upon providing techniques to support test developers by supporting more efficient source selection and evaluation (cf. Sheehan, Deane and Kostin, in this forum.) Where more constrained item types have required natural language generation, the treatment of verbal materials has been very straightforward and minimal, involving generation from verbal templates, cf. Dennis et al. 2002. The practical advantages of template-based generation are obvious: implementation …